Methods of using an imaging apparatus in augmented reality, in medical imaging and nonmedical imaging

ABSTRACT

With inventive processing making use of surface-reconstruction and capping steps, more imagery acquired by 3C cameras can be put to use in augmented reality applications, especially applications, such as medical reconstruction, in which a certain theoretical ideal fit might be wanted but can be difficult or seemingly impossible to achieve due to highly complex, irregular shapes, perimeters and surfaces involved. The inventive technology is especially useful for ongoing wound measurement and comparative analysis and characterization of a wound over time, as well as working with anatomical reconstruction. The inventive technology also extends to non-medical augmented reality applications, and provides robust data sets representing a range of real-world objects, such as zoo animals, family pets, etc. susceptible of being imaged and stored as robust data sets that provide better verisimilitude when used in gaming or other virtual-world contexts as compared to a raw data set from a camera.

FIELD OF THE INVENTION

The invention relates to augmented-reality technology.

BACKGROUND OF THE INVENTION

In an area of emerging technology, “augmented reality”, acomputer-generated image is superimposed on a user's view of the realworld, thus providing a composite view. One context of augmented realityis in medicine and healthcare. See, e.g., Winner, et al., “AugmentedReality Imaging System for Cosmetic Surgical Procedures,” US20170119471, published May 4, 2017; Gibby, et al. (Novarad Corp.),“Augmented reality viewing and tagging for medical procedures,” U.S.Pat. No. 10,010,379 issued Jul. 3, 2018.

Other contexts of augmented reality are outside of medicine orhealthcare, such as in video gaming, virtual worlds, exercise andfitness, shopping/fashion, etc. See, e.g., Rublowsky, “Augmented RealitySimulator,” US 20150260474, published Sep. 17, 2015; Parisi, “FantasySport Platform with Augmented Reality Player Acquisition,” US20180036641, published Feb. 8, 2018; Henderson, “Incentivizing foodstuffconsumption through the use of augmented reality features,” U.S. Pat.No. 10,019,628 issued Jul. 10, 2018; Bastide, et al (IBM), “Avatar-basedaugmented reality engagement,” U.S. Pat. No. 10,025,377 issued Jul. 17,2018; Laughlin, (The Boeing Co.), “Portable augmented reality,” U.S.Pat. No. 10,026,227 issued Jul. 17, 2018; Yuen, et al. (Intel Corp.),“Scene modification for augmented reality using markers withparameters,” U.S. Pat. No. 10,026,228 issued Jul. 17, 2018; Fox, et al.(Liberty Mutual Ins. Co.), “Augmented reality insurance applications,”U.S. Pat. No. 10,032,225 issued Jul. 24, 2018; Papkipos, et al.(Facebook, Inc.) “Social context in augmented reality,” U.S. Pat. No.10,032,233 issued Jul. 24, 2018; Aoki, et al. (Bally Gaming, Inc.),“Augmented reality for table games.” U.S. Pat. No. 10,046,232 issuedAug. 14, 2018; Zhang, et al. (Tencent Technology), “Method and systemfor performing interaction based on augmented reality,” U.S. Pat. No.10,049,494 issued Aug. 14, 2018; Sisbot (Toyota), “Method of groundadjustment for in-vehicle augmented reality systems,” U.S. Pat. No.10,049,499 issued Aug. 14, 2018; Morrison (3D Product Imaging Inc.),“Augmented reality e-commerce for home improvement,” U.S. Pat. No.10,049,500 issued Aug. 14, 2018.

Techniques for acquiring images that might be useable and work in onecontext are not necessarily useful in another context, or may be tooinaccurate or imprecise or prone to error, especially for medical andhealth care contexts. Various 3D imaging technology exists in medicine,but using relatively large equipment, and generally developed fordiagnosis. Improvements in 3D imaging in a direction of acquiring imagesthat will better work for augmented reality computer processing would bedesirable.

To give one example, some imaging technology that has been proposed orattempted relies on color imaging, and an easy-to-use imaging devicewithout the limitations and disadvantages of color-data processing couldbe advantageous.

SUMMARY OF THE INVENTION

The invention in one preferred embodiment provides an augmented realitymethod, comprising: operating an imaging device to acquire a 3D image;using the acquired 3D image, performing a detection algorithm thatcomprises capping or an interpolation method on a 2-dimensional grid inorder to reconstruct a surface, wherein the step is performed by acomputer or a processor and a surface reconstruction is generated; and,using the surface reconstruction, performing at least one augmentedreality processing step, virtual reality processing step, authenticreality processing step, or mixed reality processing step, wherein thestep is performed by a computer or a processor, such as, e.g. inventivemethods wherein the acquired 3D image is an image of a wound, inventivemethods wherein what is imaged in the device-operating step is otherthan a wound, inventive methods wherein a patient is imaged in thedevice-operating step, and the augmented reality is in a medicalcontext, inventive methods wherein the acquired 3D image is of anon-wound, and the augmented reality is in a non-medical, non-healthcare context; inventive methods wherein a real animal (such as, e.g., afarm animal, a household pet, a zoo animal, etc.) is imaged, and theimage is imported into a game; inventive methods wherein a body part isimaged; inventive methods wherein a healthy body part is imaged, and theimage of the healthy body part is processed to construct an image of aproposed prosthesis and/or to construct prosthesis; inventive methodswherein the processing step is in gaming; inventive methods wherein theoperating step is performed to image a patient in a first geography, andwherein the 3D image is simultaneously accessible to both a firstmedical doctor in a second geography and a second medical doctor in athird geography, wherein the first geography, second geography and thirdgeography are remote from each other; inventive methods furthercomprising a step of expanding the 3D image; inventive methods furthercomprising a step of subjecting the 3D image to contrasting; inventivemethods wherein the capping is an interpolation method on a 2D grid inorder to “reconstruct” a skin surface; inventive methods comprising astep of solving a Laplace equation with Dirichlet boundary conditions;inventive methods wherein the method steps exclude any RGB dataprocessing having been performed and without any other color-informationdata processing having been performed; inventive methods furthercomprising, using the acquired 3D image, performing a detectionalgorithm that comprises capping or an interpolation method on a2-dimensional grid in order to reconstruct a surface; inventive methodsfurther comprising performing steps of acquiring an image, previewingvideo images and selecting an Object, aiming at the center of theObject, at least one Object Scan step and at least one Object Detectionstep (such as, e.g. wherein the at least one Object Detection stepcomprises automatic detection of Object borders in the 3D depth image,Object capping and calculating an Object measurement); inventive methodsfurther comprising rendering the 3D Object model from a perpendicularcamera and generating a Z-buffer, converting the Z-buffer to depthimage, defining a region of interest U for Object detection, Objectcapping, detecting rough Object boundaries and detecting refined Objectboundaries; and other inventive methods.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an exemplary augmented-reality deviceinto which can be fitted a commercially-available 3D camera (not shown),and having incorporated therein at least one processor that performsinventive methodology.

FIG. 2 is a flow chart of method steps in an inventive embodiment ofwound measurement technology using computerized records-keeping.

FIG. 3 is a flow chart of method steps in an inventive embodiment ofwound scan and measurement.

FIG. 4 is a flow chart of method steps in an inventive embodiment ofwound detection.

FIG. 5 is a flow chart or method steps in an inventive embodiment ofwound measurements.

DETAILED DESCRIPTION OF THE INVENTION

An imaging device according to the invention is useable to acquire 3Dimages that can be subjected to computer processing steps such as invirtual reality technology, authentic reality technology, mixed realitytechnology, augmented reality technology, etc.

A significant advantage of the inventive technology is its usefulness inconnection with a patient who suffers from a wound, to compute a WoundVolume Measurement, advantageously without any ruler, grid, marker (orsuch physical object) needing to have been placed on, or near, thepatient (particularly, onto the patient wound or onto skin near thepatient wound). For patient-related usages, the invention mainlycontemplates a human patient, but also can be used in connection with aveterinary patient.

The inventive technology further is useable in connection with imaging arange of real-world objects and real-world living beings without anyphysical object needing to have been placed on, or near, the real-worldobject or living being. Post-imaging steps vary depending on what wasimaged and on the application, such as a gaming application, a limbreconstruction application, etc.

We sometimes refer herein to “touchless”, by which we mean that thepatient's wound and the wound's environ (or, in other embodiments, thereal-world object or real-world living being) is untouched by any ruler,grid, marker, 3D camera, frame enclosure holding a 3D camera, or thelike. For example, the inventive technology is useable to image a zooanimal from a safe distance, and the resulting imaged zoo animal is thenuseable, for example, in a game.

In one embodiment, an inventive augmented reality method comprises stepsof: operating an imaging device to acquire a 3D image; using theacquired 3D image, performing a detection algorithm that comprisescapping or an interpolation method on a 2-dimensional grid in order toreconstruct a surface; and performing at least one augmented realityprocessing step, virtual reality processing step, authentic realityprocessing step, or mixed reality processing step.

In the step of operating an imaging device to acquire a 3D image,examples of what is being imaged are, e.g., a real-world wound, areal-world object, a real-world living being.

In the step of operating an imaging device to acquire a 3D image, apreferred example of an imaging device is a 3D camera. Examples of a 3Dcamera for use in practicing the invention are, e.g., Real Sense 3Dcamera (manufactured by Intel); Orbbec Astra 3-D camera; ZED stereo 3-Dcamera by Stereolabs.

The invention may further be appreciated with reference to the followingexamples, without the invention being limited to these examples.

Example 1

An imaging device according to this inventive Example is useable toacquire 3D images that can be subjected to computer processing steps.For example, an imaging device that we call “Presero” was constructedaccording to this example.

Example 1.1

An imaging device was constructed as follows, according to a novelalgorithm that consists of two main parts: wound detection and woundmeasurement.

The algorithm applies to a 3D model of a human body part containing awound. The 3D model is obtained from a scan performed by an inventiveapplication. The algorithm is not applied directly to the 3D model.Instead, the generated 3D model is rendered with camera parametersproviding a good view of the wound (typically perpendicular to the woundor to the body part where the wound is), from which the algorithmacquires the Z-buffer (depth map) Z, calculated by the rendering processand the corresponding 4-by-4 projection matrix P as an input. Therendering process is based on OpenGL API (The Industry Standard for HighPerformance Graphics), and hence we use here the OpenGL terminology.

In addition, the algorithm gets a user defined outer-wound contour C asa hint for the wound location.

The algorithm does NOT use any color information.

Wound Detection

The following steps are performed.

1. Convert the Z-Buffer Z to the Depth Image D.

The conversion is given by:

${{D\left( {i,j} \right)} = \frac{P\left( {3,4} \right)}{{2{Z\left( {i,j} \right)}} - 1 + {P\left( {3,3} \right)}}},{\left( {i,j} \right) \in R},$where R={1, . . . , m}×{1, . . . , n}, m is a number of rows and n is anumber of columns in Z and D.

2. Define a Region of Interest U for Wound Detection.

We include in U all (i,j) ∈ R laying inside C, except border pixels (i=1or i=m or

j=1 or j=n) and except pixels which depth is too close to the farparameter of P, i.e.,D(i,j)>(1−α)P(3,4)/(P(3,3)+1),where α is a small positive constant.

3. Wound Capping.

We reconstruct skin surface S over the wound in order to enhance woundappearance by subtracting S from D.

(a) Calculate the First Approximation.

Since wound boundary is unknown yet, we start from the region U. Namely,we solve the following discrete Laplace equation with respect to S4S(i,j)=S(i−1,j)−S(i+1,j)−S(i,j−1)−S(i,j+1)=0if (i,j) ∈ U, andS(i,j)=D(i,j)if (i,j) ∈ R\U.(b) Iteratively Raise the Capping if Required.

There is a possibility that the surface S is situated below the woundboundary. In this case S has to be raised. Let h be a maximum value ofS−D. If, for some small tolerance threshold δ>0h>δ, then we find allpixels (i,j) ∈ U such thatS(i,j)−D(i,j)≥h−δ.Assuming that these pixels are mostly (up to the threshold δ) outsidethe wound we redefine the region U by excluding these pixels from it. Wereturn to the steps (3a) and (3b) with the updated region U. We proceedin this way until h≤δ or maximal allowed number of iterations isreached.

4. Detect a Wound.

To detect a wound we apply Chan-Vese algorithm (see T. Chan and L. Vese,Active contours without edges. IEEE Trans. Image Processing, 10(2):266-277, February 2001) to the difference F=D−S. The Chan-Veseapproach is to find among all 2-valued functions of the form

${\phi\left( {i,j} \right)} = \left\{ \begin{matrix}{{{c_{1}\mspace{14mu}{if}\mspace{14mu}\left( {i,j} \right)} \in W},} \\{{{c_{2}\mspace{14mu}{if}\mspace{14mu}\left( {i,j} \right)} \in {R\backslash W}},}\end{matrix} \right.$the one that minimizes the following energy functional,μLength(∂W)+νArea(W)+λ₁Σ_((i,j)∈W)(F(i,j)−c ₁)²+λ₂Σ_((i,j)∈R\W)(F(i,j)−c₂)²,where ∂W denotes the boundary of W, μ>0, ν≥0, λ₁>0, λ₂>0 are fixedparameters.Let W, c₁ and c₂ minimize the energy functional. We interpret W as a setof pixels belonging to the wound.

5. Correct Wound Boundary.

The wound boundary ∂W obtained in (4) is not accurate enough. It islocated somewhere on the wound walls, but not necessarily on the top ofthem. We move it to the top as described below.

Starting from each pixel (i,j) ∈ ∂W we go in the direction orthogonal to∂W and select a pixel (p(i,j), q(i,j)) located on the top of the woundwall by searching for the maximum value of the directional secondderivative of the depth image D. Our intention is to move pixels (i,j)to pixels

(p(i,j), q(i,j)), but this operation can break continuity of the woundboundary.

Denote by dist(i,j,A) the euclidean distance from the pixel (i,j) to theset of pixels A. LetΔ(i,j)=dist(i,j,W)−dist(i,j,R\W).For any t>0, the set W_(t)={(i,j) ∈ R:Δ(i,j)<t} is an uniform expansionof W with size controlled by t, W₀=W. In order to make this kind ofexpansion more flexible we replace t with a function T(i,j) which on theone hand has to be close to a constant, and on the other hand has to getvalues close to dist(p(i,j), q(i,j), W) at the pixels (p(i,j), q(i,j)).We find T as the solution of the following optimization problemΣ_(i=2) ^(m)Σ_(j=1) ^(n)[T(i,j)−T(i−1,j)]²+Σ_(i=1) ^(m)Σ_(j=2)^(n)[T(i,j)−T(i,j−1)]²+ρΣ_((i,j)∈∂W)[T(p(i,j),q(i,j))−dist(p(i,j),q(i,j),W)]²→min,where ρ>0 is a constant parameter. Finally, we declareW*={(i,j)∈R: Δ(i,j)≤T(i,j)}

as a set of the wound pixels,

Wound Measurements

Formulas for calculating wound volume, maximal depth, area, perimeter,length and width are set forth below. Note that the last 4 measurementsare calculated for wound projection onto a plane parallel to the cameraimage plane.

In order to calculate wound volume we perform capping again as describedin (3a) using W* instead of U. Let S* be the result. We clamp it asfollowsS*=min(S*,D).Then

${{WoundVolume} = {\frac{4}{3{{mnP}\left( {1,1} \right)}{P\left( {2,2} \right)}} \cdot {\sum\limits_{{({i,j})} \in W}{.\left( {{D\left( {i,j} \right)}^{3} - {S^{*}\left( {i,j} \right)}^{3}} \right)}}}},{{WoundMaximalDepth} = {\max{\left\{ {{{D\left( {i,j} \right)} - {S^{*}\left( {i,j} \right)}},{\left( {i,j} \right) \in W^{*}}} \right\}.}}}$Tracing the wound boundary ∂W* we write down all pixels belonging to ∂W*as a sequence (i₁,j₁), (i₂,j₂), . . . , (i_(N),j_(N)). Let Q be theinverse matrix of P and let for each k=1, . . . , N,

${X_{k} = \frac{{{Q\left( {1,1} \right)}x_{k}} + {Q\left( {1,4} \right)}}{{{Q\left( {4,3} \right)}z_{k}} + {Q\left( {4,4} \right)}}},{Y_{k} = \frac{{{Q\left( {2,2} \right)}y_{k}} + {Q\left( {2,4} \right)}}{{{Q\left( {4,3} \right)}z_{k}} + {Q\left( {4,4} \right)}}},{where}$${x_{k} = {{\left( {2\text{/}n} \right)\left( {j_{k} - 0.5} \right)} - 1}},{y_{k} = {{{- \left( {2/m} \right)}\left( {i_{k} - 0.5} \right)} + 1}},{z_{k} = {{- {P\left( {3,3} \right)}} + {\frac{P\left( {3,4} \right)}{D\left( {i_{k},j_{k}} \right)}.}}}$Put, in addition, X₀=X_(N),Y₀=Y_(N) and Y_(N+1)=Y₁.ThenWoundArea=|Σ_(k=1) ^(N) X _(k)(Y _(k+1) −Y _(k−1))|,WoundPerimeter=Σ_(k=1) ^(N)√{square root over ((X _(k) −X _(k−1))²+(Y_(k) −Y _(k−1))².)}Assuming that a human body orientation is defined by an angle θ, woundlength and width are given byWoundLength=max{X _(k) cos θ+Y _(k) sin θ,1≤k≤N}−min{X _(k) cos θ+Y _(k)sin θ,1≤k≤N},WoundWidth=max{−X _(k) sin θ+Y _(k) cos θ,1≤k≤N}−min{−X _(k) sin θ+Y_(k) cos θ,1≤k≤N}.

REFERENCES

-   1. T. Chan and L. Vese, Active contours without edges. IEEE Trans.    Image Processing, 10 (2):256-277, February 2001.

Example 1A

Optimal values for algorithm parameters in Example 1 are determined bytesting the system on phantom wounds and other forms made fromplasticine. For α (a small positive constant), 0.01 was chosen.

Example 1B

In this example, when an inventive device used according to an aboveexample, an image was ready to view within 10 seconds of cameraoperation.

Example 1C

In this example, when an inventive device was used according to an aboveexample, after a scan was completed, a 3D image was displayed to a user,and the displayed 3D image was subject to being manipulated by a fingerof the user.

Example 1D

In this example according to Example 1C, a user manipulated a woundimage on screen with the user's finger, including, the user lookedbehind and under a wound image on screen.

Example 1E

Referring to FIG. 2, in this Example, method steps are performed of:creating 200 a New Patient record or selecting 201 an Existing Patientrecord; presenting 202 a gallery of the patient's wounds; creating 203 anew Wound record or selecting 204 an existing Wound record; performingWound Scan & Measurement 205; adding 206 the scan to Wound ScansHistory; presenting 207 Wound Volume trend line, Wound Measurement PerScan, and Total Volume Reduction from first scan.

Example 1F

Referring to FIG. 2, optionally steps of adding 203A wound location andtype (to the Wound Record, and/or adding/editing 200A patient details tothe Patient Record, are performed.

Example 1F Wound Scan & Measurement

Referring to FIG. 3, in this Example, method steps are performed of:Image Acquisition 300 using a 3D depth and 2D camera module; previewing301 video images and selecting a wound to measure; aiming 302 at thecenter of the wound, from a proper distance; starting 303 scanning;manipulating 304 the camera around the wound center; stopping 305scanning; a step 307, performed by an operator, of marking a woundcontour as a first estimation and defining wound-body orientation;automatic detection 308 of wound borders in the 3D depth image; woundcapping 309 (comprising estimating the optimal upper closure (i.e., cap)for the wound); calculating 310 wound measurement (comprising measuringthe volume beneath the cap, wound circumference, width, length, maximumdepth, and area).

Steps 303, 304, 305 are referred to as Wound Scan 306 steps.

Steps 308, 309, 310 are referred to as Wound Detection 311 steps.

Example 1G

Referring to FIG. 3, optionally the operator is allowed to manuallycorrect 308A bound borders.

Example 1H

Referring to FIG. 3, optionally real-time wound tracking and datacollection are output in an outputting step 306A.

Example 1I

Referring to FIG. 3, optionally a 3D model of the wound is generated ina generating step 306B.

Example 1J

Referring to FIG. 3, optionally the 3D model of Example 2.8B ispresented to the operator in a displaying step 311A.

Example 1K Wound Detection

Referring to FIG. 4, in this Example, steps are performed of: a step 401of rendering the 3D wound model from a perpendicular camera andgenerating Z-buffer (using OpenGL); converting 402 the Z-buffer to depthimage; defining 403 a region of interest U for wound detection; woundcapping 404 (comprising reconstructing skin surface over the wound);rough wound boundaries detection 405; and refined wound boundariesdetection 406.

Example 1L Wound Measurements

Referring to FIG. 5, in this Example, steps are performed of: measuring501 distances from capping to wound floor; calculating 502 volume bysumming distances in all pixels inside the wound; calculating 503maximum depth (max distances); summating 504 perimeter length equalingtotal length of detected wound boundaries; calculating 505 wound areafrom detected wound boundaries; calculating 506 max wound length & widthby aligning the wound contour to body angle; and calculating 507presented area as Max length×Max width.

Example 2

In this example, an imaging device of Examples 1-1A is used to imagesomething other than a wound.

Example 3

In this example, when an imaging device was used according to any ofExamples 1-2, an image was ready to view within 10 seconds of cameraoperation.

Example 4

In this example, when an imaging device was used according to any ofExamples 1-3, after a scan was completed, a 3D image was displayed to auser, and the displayed 3D image was subject to being manipulated by afinger of the user.

Example 4A

In this example according to Example 4, a user manipulated a wound imageon screen with the user's finger, including to look behind and under awound image on screen.

Example 5

An imaging device for imaging an Object which may be other than a woundis constructed. A 3D model of the Object is obtained from a scanperformed by the imaging application. The algorithm is not applieddirectly to the 3D model. Instead, the generated 3D model is renderedwith camera parameters providing a good view of the Object (typicallyperpendicular to the Object or to the region where the Object is), fromwhich the algorithm acquires the Z-buffer (depth map) Z, calculated bythe rendering process and the corresponding 4-by-4 projection matrix Pas an input. The rendering process is based on OpenGL API.

In addition, the algorithm gets a user defined outer-Object contour C asa hint for the Object location.

Example 5.1 (Object Detection)

This Object-Detection part of the algorithm is represented by thefollowing steps.

1. Convert the Z-buffer Z to the Depth Image D.

The conversion is given by

${{D\left( {i,j} \right)} = \frac{P\left( {3,4} \right)}{{2{Z\left( {i,j} \right)}} - 1 + {P\left( {3,3} \right)}}},{\left( {i,j} \right) \in R},$Where R=[1, . . . , m]×[1, . . . , n], m is a number of rows and n is anumber of columns in Z And D.2. Define a Region of Interest U for Object Detection.

We include in U all (i,j) ∈ R laying inside C, except border pixels (i=1or i=m or j=1 or j=n) and except pixels which depth is too close to thefar parameter of P, i.e.,D(i,j)>(1−α)P(3,4)/(P(3,3)+1),Where α is a small positive constant.3. Object Capping.

We reconstruct skin surface S over the Object in order to enhance Objectappearance by subtracting S from D.

(a) Calculate the First Approximation.

Because Object boundary is unknown yet, we start from the region U.Namely, we solve the following discrete Laplace equation with respect toS4S(i,j)−S(i−1,j)−S(i+1,j)−S(i,j−1)−S(i,j+1)=0if (i,j) ∈ U, andS(i,j)=D(i,j)if (i,j) ∈ R\U.(b) Iteratively Raise the Capping if Required.

There is a possibility that the surface S is situated below the Objectboundary. In this case S has to be raised. Let h be a maximum value ofS−D. If, for some small tolerance threshold δ>0h>δ, then we find allpixels (i,j) ∈ U such thatS(i,j)−D(i,j)≥h−δ.Assuming that these pixels are mostly (up to the threshold δ) outsidethe Object we redefine the region U by excluding these pixels from it.We return to the steps (3a) and (3b) with the updated region U. Weproceed in this way till h≤δ or maximal allowed number of iterations isreached.4. Detect an Object.

To detect an Object we apply Chan-Vese algorithm [1] to the differenceF=D−S. The Chan-Vere approach is to find among all 2-valued functions ofthe form

${\phi\left( {i,j} \right)} = \left\{ \begin{matrix}{{{c_{1}\mspace{14mu}{if}\mspace{14mu}\left( {i,j} \right)} \in W},} \\{{{c_{2}\mspace{14mu}{if}\mspace{14mu}\left( {i,j} \right)} \in {R\backslash W}},}\end{matrix} \right.$the one that minimizes the following energy functional,μLength(∂W)+νArea(W)+λ₁Σ_((i,j)∈W)(F(i,j)−c ₁)²+λ₂Σ_((i,j)∈R\W)(F(i,j)−c₂)²,Where ∂W denotes the boundary of W, μ>0, ν≥0, λ₁>0, λ₂>0 are fixedparameters.

Let W, c₁ and c₂ minimize the energy functional. We interpret W as a setof pixels belonging to the wound.

5. Correct Object Boundary.

The Object boundary ∂W obtained in (4) is not accurate enough. It islocated somewhere on the Object walls, but not necessarily on the top ofthem. We move it to the top as described below.

Starting from each pixel (i,j) ∈ ∂W we go in the direction orthogonal to∂W and select a pixel (p(i,j), q(i,j)) located on the top of the woundwall by searching for the maximum value of the directional secondderivative of the depth image D. Our intention is to move pixels (i,j)to pixels (p(i,j), q(i,j)), but this operation can break continuity ofthe Object boundary.

Denote by dist(i,j,A) the euclidean distance from the pixel (i,j) to theset of pixels A. LetΔ(i,j)=dist(i,j,W)−dist(i,j,R\W).

For any t>0, the set W_(t)={(i,j) ∈ R:Δ(i,j)<t} is an uniform expansionof W with size controlled by t, W₀=W. In order to make this kind ofexpansion more flexible we replace t with a function T(i,j) which on theone hand has to be close to a constant, and on the other hand has to getvalues close to dist(p(i,j), q(i,j),W) at the pixels (p(i,j), q(i,j)).

We find T as the solution of the following optimization problemΣ_(i=2) ^(m)Σ_(j=1) ^(n)[T(i,j)−T(i−1,j)]²+Σ_(i=1) ^(m)Σ_(j=2)^(n)[T(i,j)−T(i,j−1)]²+ρΣ_((i,j)∈∂W)[T(p(i,j),q((i,j))=dist(p(i,j),q(i,j),W)]²→min,where ρ>0 is a constant parameter. Finally, we declareW*={(i,j)∈R:Δ(i,j)≤T(i,j)}as a set of the Object pixels.

Example 5.2 Object Measurements

In this part we present formulas for calculating Object volume, maximaldepth, area, perimeter, length and width. The last 4 measurements arecalculated for Object projection onto a plane parallel to the cameraimage plane.

In order to calculate Object volume we perform capping again asdescribed in (3a) using W* instead of U. Let S* be the result. We clampit as follows

${X_{k} = \frac{{{Q\left( {1,1} \right)}x_{k}} + {Q\left( {1,4} \right)}}{{{Q\left( {4,3} \right)}z_{k}} + {Q\left( {4,4} \right)}}},$

Tracing the Object boundary ∂W* we write down all pixels belonging to∂W* as a sequence (i₁,j₁), (i₂,j₂), . . . , (i_(N),j_(N)). Let Q be theinverse matrix of P and let for each k=1, . . . , N,

S^(*) = min (S^(*), D).Then${{{Object}\mspace{14mu}{Volume}} = {\frac{4}{3{{mnP}\left( {1,1} \right)}{P\left( {2,2} \right)}} \cdot {\sum\limits_{{({i,j})} \in W}{.\left( {{D\left( {i,j} \right)}^{3} - {S^{*}\left( {i,j} \right)}^{3}} \right)}}}},{{{Object}\mspace{14mu}{MaximalDepth}} = {\max{\left\{ {{{D\left( {i,j} \right)} - {S^{*}\left( {i,j} \right)}},{\left( {i,j} \right) \in W^{*}}} \right\}.}}}$

${Y_{k} = \frac{{{Q\left( {2,2} \right)}y_{k}} + {Q\left( {2,4} \right)}}{{{Q\left( {4,3} \right)}z_{k}} + {Q\left( {4,4} \right)}}},{where}$${x_{k} = {{\left( {2\text{/}n} \right)\left( {j_{k} - 0.5} \right)} - 1}},{y_{k} = {{{- \left( {2/m} \right)}\left( {i_{k} - 0.5} \right)} + 1}},{z_{k} = {{- {P\left( {3,3} \right)}} + {\frac{P\left( {3,4} \right)}{D\left( {i_{k},j_{k}} \right)}.}}}$Put, in addition, X₀=X_(N), Y₀=Y_(N) and Y_(N+1)=Y₁.ThenObjectArea=|Σ_(k=1) ^(N) X _(k)(Y _(k+1) −Y _(k−1))|,ObjectPerimeter=Σ_(k=1) ^(N)√{square root over ((X _(k) −X _(k−1))²+(Y_(k) −Y _(k−1))²)}.Assuming that a Locality orientation is defined by an angle θ, Objectlength and width are given byObjectLength=max{X _(k) cos θ+Y _(k) sin θ,1≤k≤N}−min{X _(k) cos θ+Y_(k) sin θ,1≤k≤N},ObjectWidth=max[−X _(k) sin θ+Y _(k) cos θ,1≤k≤N]−min[−X _(k) sin θ+Y_(k) cos θ,1≤k≤N].

Example 5.3

An example of an Object in Example 5.1 (Object Detection) above is anintact body part that is being 3D-imaged and the 3D image is processedin at least one prosthesis-modeling or prosthesis-construction steps.

The above described embodiments are set forth by way of example and arenot limiting. It will be readily apparent that obvious modification,derivations and variations can be made to the embodiments. The claimsappended hereto should be read in their full scope including any suchmodifications, derivations and variations.

What is claimed is:
 1. An augmented or virtual reality method of woundimaging and reconstruction, comprising: operating an imaging device toacquire a 3D image of a wound that is real; producing a depth image fromthe 3D image; detecting a wound from the depth image, includingproducing a preliminary wound boundary formed of pixels, producing afinal wound boundary from the preliminary wound boundary, including, foreach pixel in the preliminary wound boundary, searching for a maximumvalue of a directional second derivative of the depth image along adirection orthogonal to the preliminary wound boundary, setting a pixelof the final wound boundary to coordinates corresponding with themaximum value, subject to a size control function to avoid breakingcontinuity of the final wound boundary; and using the final woundboundary, performing at least one augmented reality processing step,virtual reality processing step, authentic reality processing step, ormixed reality processing step, wherein the step is performed by acomputer or a processor.
 2. The method of claim 1, wherein the operatingstep comprises imaging a human patient.
 3. The method of claim 1,wherein the operating step comprises imaging a real animal.
 4. Themethod of claim 3, wherein the real animal is selected from the groupconsisting of: a farm animal; a household pet; a zoo animal.
 5. Themethod of claim 1, wherein the operating step is performed to image apatient in a first geographic location, and wherein the 3D image issimultaneously accessible to both a first medical professional in asecond geographic location and a second medical professional in a thirdgeographic location, wherein the first geographic location, secondgeographic location, and third geographic location are remote from eachother.
 6. The method of claim 1, comprising a step of solving a Laplaceequation with Dirichlet boundary conditions.
 7. The method of claim 1,wherein the method steps exclude any RGB data processing having beenperformed and without any other color-information data processing havingbeen performed.
 8. The method of claim 1, further comprisingconstructing a virtual skin surface using the acquired 3D image usingcapping or an interpolation method on a 2-dimensional grid.
 9. Themethod of claim 1, further comprising calculating a measurement from thefinal wound boundary.
 10. The method of claim 1, further comprising:generating a Z-buffer, wherein the depth image is produced by aconversion of the Z-buffer; defining a region of interest U for thewound detection step.